Customized program insertion system

ABSTRACT

A customized consumer advertisement module and process are provided for inserting a user selected object into a user selected category of program. The user stores model objects locally at the user site, or communicates the model objects to a remotely located broadcaster for storage at a broadcaster site. The user selects categories of programs in which the model objects, or portions thereof, will be inserted. The customized consumer advertisement module inserts the model objects, or portions thereof, into the selected categories of programs.

FIELD OF THE INVENTION

The present invention relates to the field of communications. Moreparticularly, the present invention relates to gathering, managing andinserting customized program information in communication systems.

BACKGROUND INFORMATION

User expectation has continued to grow concurrently with the advent ofnew technologies. Expectations for greater and more personalizedservices have developed in a variety of service industries with respectto inserting images or other visual information into another displayedimage, replacing at least a portion of that display image. For example,U.S. Pat. No. 4,731,743 to BLANCATO describes a “Method and Apparatusfor Displaying Hairstyles.” In an effort to provide a user with apreview of how a particular hairstyle will appear on the user, i.e.,without having to go through the actual haircut and hairstyling,BLANCATO stores a plurality of hairstyles on a disc associated with acomputer. Subsequently, BLANCATO captures an image of the user andoutlines the user's face. He then makes a number of modifications to theimage, including overlaying particular hairstyles. BLANCATO thendisplays the overlayed image to the user for hairstyle assessment.

U.S. Pat. No. 6,624,843 to LENNON, entitled “Customer Image Capture andUse Thereof in a Retailing System,” describes an image capture systemused in a retailing environment. Particularly, LENNON describes a systemthat captures an image of a customer at a retailer's place of business.Subsequently, when the customer is in close proximity to an imagedisplay area within the retailer's place of business, a composite imageincluding the customer's captured image and a reference image may beprovided. The composite image may include full motion video or stillimages. Thus, a customer is provided with an opportunity to virtuallyassess particular merchandise without having to try on, for example, anapparel item.

U.S. Pat. No. 7,015,978 to JEFFERS et al., entitled “System And MethodFor Real Time Insertion Into Video With Occlusion On Areas ContainingMultiple Colors,” describes real-time insertion of indicia such as, forexample, an advertisement, into a live or taped broadcast. JEFFERS etal. provide for insertion of indicia onto surfaces such as tenniscourts, walls, grass or turf surfaces, or other surfaces shown duringtelevised sporting events. To accomplish indicia insertion, JEFFERS etal. obtain video images from a camera, digitize the images, sample thedigitized images into sampled pixels, and obtain a plurality ofbackground colors for the sampled pixels. JEFFERS et al. then assign anopacity value to each pixel in the indicia based on whether the color ofa positionally corresponding pixel in the video image is the same coloras one of the plurality of background colors. Pixels are then displayedin the video image in dependence on whether the positionallycorresponding pixel in the video image is the same color as one of thebackground colors.

Progress in computer processor speeds has significantly expanded thepossible applications. It is not uncommon for ordinary personalcomputers to possess microprocessors that operate at clock cycles of 3.8GHz or higher. Moreover, storage capacity has increased dramatically,where it is not uncommon to find ordinary personal computers thatpossess 200 Gigabyte hard-drives, which may be further expanded withoutany clearly defined limit.

A need exists for customized signal insertion and/or overlay intelecommunications systems, such as, for example, television and radiobroadcasts. Particularly, a need exists for customized insertion and/oroverlay of user defined images, sounds and other indicia in receivedsignals on electronic devices such as, for example, televisions,computers, telephones, personal data assistants (PDAs), portablecomputers, radios, satellite receivers, and the like.

SUMMARY OF THE INVENTION

According to an aspect of the invention, an image insertion apparatus isprovided for inserting a model object into at least one program. Theobject may comprise an image, a sound, or a combination of an image anda sound. The object insertion apparatus may be provided at a user site,a transmitter site, or a combination of the user site and transmittersite. The program may comprise a multimedia content signal that may betransmitted from the user site, the transmitter site, or a combinationof the user site and transmitter site.

The object insertion apparatus includes a memory, a selector, acommunicator and an inserter. The memory stores one or more modelobjects. The selector receives a selection of the one or more modelobjects and a selection of one or more categories of program. At leastone of the categories of program includes the at least one program.

The communicator provides communication between the object insertionapparatus and a user. The communicator is configured to receive theselected one or more model objects and the selected one or morecategories of program. The selected one or more model objects areprovided to the inserter, which inserts and/or overlays the selected oneor more model objects into the at least one program based on theselected one or more categories of program.

The object insertion apparatus further includes an object extractor, afeature extractor, an object processor, an encryptor, an encoder, aformat converter, a decryptor, and a decoder. The decryptor decrypts areceived multimedia signal, including the at least one program, when thereceived signal is determined to be encrypted. The decoder decodes thereceived multimedia signal, including the at least one program, andoutputs a decoded multimedia signal. The decoded multimedia signal isprovided to the format converter, which converts the decoded multimediasignal to a base composite signal, including the at least one program.The base composite signal includes an object field signal and an audiosignal, where the object field signal and the audio signal may be the atleast one program.

The object field signal is provided to the extractor, which extracts aportion of the object field signal from the at least one program. Theobject extractor generates a histogram for the extracted portion of theobject field signal by transforming the extracted portion of the objectfield signal from a time domain to a frequency domain. The extractedportion of the object field signal is provided to the feature extractor,which extracts a feature from the extracted portion of the object fieldsignal. The feature extractor generates at least one motion vector forthe extracted portion of the object field signal. The extracted portionof the object field signal and the extracted feature from the extractedportion of the object field signal are provided to the object processor.

The object processor modifies the selected one or more model objectsbased on the extracted portion of the object field signal, and theextracted feature from the extracted portion of the object field signal,to generate one or more modified model objects. The object processorgenerates at least one insertion object from the one or more modifiedmodel objects based on the extracted portion and the generatedhistogram. The object processor inserts and/or overlays the one or moremodified model objects in the object field signal to generate a modifiedcomposite signal, including a modified object field signal, whichincludes at least one modified program. The object processor may selectat least one of an insertion and overlay mode based on a received modesignal.

The format converter converts the modified base composite signal,including the modified object field signal, which includes the at leastone modified program, to a modified multimedia signal. The encoderencodes the modified multimedia signal to provide an encoded modifiedmultimedia signal. Further, depending on whether encryption isnecessary, the encryptor encrypts the encoded modified multimediasignal.

According to another aspect of the invention, an object insertion methodis provided for inserting a selected object into at least one program.The at least one program may be associated with one of a plurality ofcategories of programs.

According to the method, one or more model objects are received andstored. At least one category of program is selected for which insertionor overlaying is desired to be performed. Further, at least one of themodel objects is selected for insertion, or overlaying, in the at leastone category of programs. The at least one category of programs includesthe at least one program.

Communication is carried out between an object insertion apparatus and auser. The communication includes receiving the selected one or moremodel objects and the selected at least one category of program. Theselected one or more model objects are inserted, or overlayed, in the atleast one program based on the selected one or more categories ofprogram.

Further, when a received signal is determined to be encrypted, thereceived signal is decrypted. The decrypted signal, or the receivedsignal if it is received in decrypted form, is decoded to provide adecoded multimedia signal, which includes the at least one program. Thedecoded multimedia signal is format converted to generate a basecomposite signal, which includes the at least one program. The basecomposite signal includes an object field signal and an audio signal,where the object field signal and the audio signal may be the at leastone program.

A portion of the object field signal is extracted from the at least oneprogram. During the portion extraction process, a histogram may begenerated for the extracted portion of the object field signal bytransforming the extracted portion of the object field signal from atime domain to a frequency domain. A feature is extracted from theextracted portion of the object field signal to provide at least oneextracted feature. During the process of feature extraction, at leastone motion vector may be generated for the extracted portion of theobject field signal. The extracted portion of the object field signaland the at least one extracted feature from the extracted portion of theobject field signal are forwarded for object processing.

Object processing is provided for modifying the selected one or moremodel objects based on the extracted portion of the object field signal,and the extracted feature from the extracted portion of the object fieldsignal, to generate one or more modified model objects. The objectprocessing generates at least one insertion object from the one or moremodified model objects based on the extracted portion and the generatedhistogram. The object processing inserts, or overlays, the one or moremodified model objects in the object field signal to generate a modifiedcomposite signal, including a modified object field signal, whichincludes the at least one modified program.

Modified base composite signal, including the modified object fieldsignal, which includes the at least one modified program, are formatconverted to a modified multimedia signal. The modified multimediasignal may be encoded to provide an encoded modified multimedia signal.Further, depending on whether encryption is necessary, the encodedmodified multimedia signal may be encrypted to provide an encryptedencoded modified multimedia signal.

According to yet another aspect of the invention, a computer readablemedium for storing a program that inserts a selected object into atleast one program, the at least one program being associated with one ofa plurality of categories of programs, is provided. The medium includesa storing code segment for storing one or more model objects; a categoryidentifying code segment for identifying a category of program from theplurality of categories of programs; a model object selecting codesegment for selecting a model object from the stored one or more modelobjects based on the identified category of program; an inserting codesegment for inserting the selected model object in the at least oneprogram. The medium further includes a receiving code segment forreceiving the selected model object and the identified category ofprogram from a user site.

Further, the medium includes a portion extracting code segment forextracting a portion of an object field signal; a feature extractingcode segment for extracting a feature from the extracted portion of theobject field signal; a modifying code segment for modifying the selectedmodel object based on the extracted portion of the object field signaland the extracted feature from the extracted portion of the object fieldsignal; a histogram generating code segment for generating a histogramfor the extracted portion of the object field signal; and a motionvector generating code segment for generating at least one motion vectorfor the extracted portion of the object field signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is further described in the detailed descriptionthat follows, by reference to the noted drawings by way of non-limitingexamples of embodiments of the present invention, in which likereference numerals represent similar parts throughout the several viewsof the drawings:

FIG. 1 illustrates an embodiment of a telecommunications system,according to an aspect of the present invention;

FIG. 2 illustrates an exemplary embodiment of an image insertionapparatus according to an aspect of the present invention;

FIG. 3 illustrates an exemplary embodiment of a customized consumeradvertisement module according to an aspect of the present invention;

FIG. 4 illustrates an exemplary multi-image capture scheme forgenerating a model image according to an aspect of the presentinvention;

FIGS. 5A through 5F are exemplary images captured by the exemplarymulti-image capture scheme of FIG. 4, according to an aspect of thepresent invention;

FIG. 6 is exemplary image showing a portion of a model image to beextracted according to an aspect of the present invention; and

FIG. 7 is a flow diagram of an exemplary process for inserting aselected image in a selected category of advertisement according to anaspect of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

An embodiment of the invention is shown in FIG. 1. The system 100includes a display device 110 for displaying moving or still images. Thedisplay device 110 may be, for example, a cathode ray tube (CRT), aliquid crystal display (LCD), a light emitting diode (LED) display, aplasma display panel (PDP), an organic light emitting diode (OLED)display, a surface-conduction electron-emitter display (SED), a carbonnanotube (CNT), a nanocrystal display (NCD), or the like. Moreover, thedisplay device may include user-mounted devices such as, for example, ahead-mount display, and/or a three-dimensional display such as, forexample, a holographic display.

Further, the display device 110 can be a portable computer device suchas, for example, a personal data assistant (PDA), telephone device,portable music player, portable game device, or any other portablecomputer device capable of displaying still and/or moving images.

The display 110 may include audio speakers (not shown), which areintegrally configured in the display 110. However, speakers may beprovided separately from the display 110, as is known in the relevantarts.

An image signal is provided to the display 110 via a Settop Box 105. TheSettop Box 105 receives multimedia signals from a Broadcaster 120 viawired and/or wireless communications links. In the exemplary embodiment,the multimedia signals are received from a network 115, which may be alocal area network, a wide area network, the Internet, or anycombination of networks, including wired and/or wireless networks, asthe skilled artisan will readily appreciate, without departing from thespirit and/or scope of the invention. The Settop Box 105 converts thereceived multimedia signal into a standard video image signal and/or astandard audio signal to be reproduced on the display 110. For example,the Settop Box 105 may be configured to decrypt the received multimediasignals and forward the signals as unencrypted, standard video imageand/or audio signals capable of reproduction on display 110. The SettopBox 105 may be configured, for example, as device 200 shown in FIG. 2(described in greater detail below), or it may be an off-the-shelf itemavailable through, for example, local cable companies.

Alternatively, the video image and/or audio signals may be provideddirectly to the image display 110 from the network 115. In this case,the device 200 may be provided at the Broadcaster 120 or within theimage display 110 as an integral device, or as a standalone devicelocated remotely, but accessible to Broadcaster 120.

The multimedia signals are forwarded, or routed, to the Settop Box 105via network 115 from a Broadcaster 120. A portion of Broadcaster 120 maybe configured as device 200 shown in FIG. 2, as stated above.Alternatively, the Broadcaster 120 may include an existing configurationsuch as, for example, that of an existing cable television broadcaster.The Broadcaster 120 is linked to an external Database 125 for storing,managing and retrieving multimedia data to be broadcast by Broadcaster120.

In an embodiment, Settop Box 105 is configured as device 200 shown in,for example, FIG. 2 in broken lines. However, as stated above, thedevice 200 may be integrated into the display device 110, theBroadcaster 120, or provided as a separate stand-alone unit at theBroadcaster 120 site or the user site without departing from the scopeand/or spirit of the invention.

The device 200 is a Custom Consumer Advertisement Module (CCAM), whichmay be a standalone device or embodied in another device. CCAM 200includes a plurality of ports for communicating with other devices, suchas multimedia devices 205-225. The CCAM 200 may be coupled to devices205-225 through wired links, such as, for example, universal serialbuses (USB), or through wireless links, such as, for example, optical orelectromagnetic frequency communications devices (for example, infrareddiode transceivers, radio frequency transceivers, etc.).

The CCAM 200 may be coupled to an external digital video disc (DVD)player 205 through wired or wireless communication linkage to receivemultimedia signals from the DVD 205 for display on display 110.Alternatively, the DVD 205 may be internal to the Settop Box 105. TheDVD 205 reads multimedia signals from a source DVD, or alternatively, aBlue-ray disc (BD), compact disc (CD), Super Audio compact disc (SACD),or the like, and provides the multimedia signals to a random accessmemory (RAM) 235. However, the RAM 235 may be bypassed and themultimedia signals from the DVD 205 may be forwarded to a Database 245for storage.

The CCAM 200 may be configured to be coupleable to a digital camera 210,for retrieving images stored in the camera 210. The CCAM 200 may beconfigured for real-time image capture using camera 210. The camera 210may be internal to the Settop Box 105, or it may be provided as aseparate, external device that is coupleable to the Settop Box 105.

Further, the CCAM 200 may be coupled to a computer device 215 such as,for example, a personal computer, a desktop computer, a workstation, alaptop computer, a notebook computer, a portable computer, a tabletcomputer, or the like. Multimedia signals stored in or generated by thecomputer device 215 may be retrieved and processed by the CCAM 200. Thecomputer device 215 may be integral to the Settop Box 105, or it may beprovided externally as shown in, for example, FIG. 2.

Further, the CCAM 200 may be coupled to a video camera 220 for capturingmoving images. Moving images may be downloaded from the video camera220, or may be captured real-time by the video camera 220 under controlof the CCAM 200. The video camera may be integral to the Settop Box 105,or it may be provided externally as shown in, for example, FIG. 2.

Further, the CCAM 200 may be coupled to a personal device (PD) 225 suchas, for example, a personal data assistant (PDA), a telephone device, orany other computer device capable of providing multimedia signals. TheCCAM 200 communicates with PD 225 to exchange multimedia signals. Forexample, the CCAM 200 may download multimedia signals, such as, forexample, still images, moving images, audio segments, computer programs,including computer instructions, and/or global position system (GPS)information from the CCAM 200 to the PD 225. Moreover, the PD 225 mayupload multimedia signals, such as, for example, still images, movingimages, audio segments, computer programs, including computerinstructions, GPS information, and the like, to the CCAM 200.

In an embodiment, the CCAM 200 includes a Controller 230, Random AccessMemory (RAM) 235, Read Only Memory (ROM) 240, Database 245, Input/Output(I/O) Interface 250, and Processor 255 for carrying out aspects of theinvention.

The Controller 230 controls task management of the components of CCAM200. In a particular embodiment, the CCAM may include a disc drive unit(not shown), including a computer-readable medium in which one or moresets of instructions, e.g., software and/or segments of code, may beembedded. Further, the instructions may embody one or more of themethods or logic as described herein. In a particular embodiment, theinstructions may reside completely, or at least partially, within theController 230, the RAM 235, the ROM 240, the Processor 255 and/or theDatabase 245 during execution by the CCAM 200. The RAM 235, the Database245 and the Processor 255 also may include computer-readable media. TheCCAM 200 operates under control of the Controller 230, to performcustomized advertisement insertion.

In alternative embodiments, dedicated hardware implementations for theCCAM 200, such as, for example, application specific integratedcircuits, programmable logic arrays and other hardware devices, can beconstructed to implement one or more of the methods described herein.Applications that may include various embodiments can broadly include avariety of electronic and computer systems. One or more embodimentsdescribed herein may implement functions using two or more specificinterconnected hardware modules or devices with related control and datasignals that can be communicated between and through the modules, or asportions of an application-specific integrated circuit. Accordingly, thepresent system encompasses software, firmware, and hardwareimplementations.

The methods described herein may be implemented by software programsexecutable by a computer system. Further, in an exemplary, non-limitedembodiment, implementations can include distributed processing,component/object distributed processing, and parallel processing.Alternatively, virtual computer system processing can be constructed toimplement one or more of the methods or functionality as describedherein.

The term “computer-readable medium” includes a single medium or multiplemedia, such as a centralized or distributed database, and/or associatedcaches and servers that store one or more sets of instructions. The term“computer-readable medium” shall also include any medium that is capableof storing, encoding or carrying a set of instructions for execution bya processor, or that cause a computer system to perform any one or moreof the methods or operations disclosed herein.

The computer-readable medium may further include a solid-state memory,such as a memory card, that houses one or more non-volatile read-onlymemories. Further, the computer-readable medium can be a random accessmemory or other volatile re-writable memory. Additionally, thecomputer-readable medium can include a magneto-optical or opticalmedium, such as a disc or tapes or other storage device to capturecarrier wave signals such as a signal communicated over a transmissionmedium. A digital file attachment to an e-mail or other self-containedinformation archive or set of archives may be considered a distributionmedium that is equivalent to a tangible storage medium. Accordingly, thedisclosure is considered to include any one or more of acomputer-readable medium or a distribution medium and other equivalentsand successor media, in which data or instructions may be stored.

Using, for example, CCAM 200 shown in FIG. 2, a process for customizedadvertisement insertion may be provided. The system of FIG. 2 can alsooperate as various elements within the system. For example, a programimplementing the disclosure may be loaded and executed on one or morecomputers.

An exemplary, non-limiting embodiment of the Processor 255 (shown inFIG. 2) is provided in FIG. 3. The Processor 255 may include anInput/Output (I/O) Interface module 310 for receiving and sendinginformation; a Decryptor/Encryptor module 315 for decrypting/encryptingsignals received from, or to be sent through the I/O Interface module310; a Decoder/Encoder module 320 for decoding and/or encoding signalsreceived, or to be sent through I/O Interface module 310; a FormatConverter module 325 for converting signals to an appropriate format fordisplay and/or signal processing; a Memory module 330 for storinginformation; a Controller module 335 for controlling task management inthe Processor 255; an Image Extractor module 340 for extracting portionsof an image signal; a Feature Extractor module 345 for extractingfeatures from the extracted portion of the image signal; and an ImageProcessor 355 for processing a multimedia signal 302 received by the I/OInterface module 310. The modules communicate via a bus 365.

In an embodiment, the I/O Interface 310 receives the multimedia signal302 from the network 115. The I/O Interface 310 forwards the receivedmultimedia signal 302 to the Decryptor/Encryptor module 315 and/or theDecoder/Encoder module 320, via the bus 365.

Alternatively, the I/O Interface 310 may be integrated with the I/OInterface 250 (shown in FIG. 2), or eliminated entirely and only the I/OInterface 250 used instead. Moreover, the bus 365 may be eliminated andmodules 310 to 355 may be coupled via bus 260 (shown in FIG. 2).

The received multimedia signal 302 is forwarded to theDecryptor/Encryptor module 315. The Decryptor/Encryptor module 315decrypts the multimedia signal 302, as is well known in the art, andforwards the decrypted signal to Decoder/Encoder module 320. However, ifthe multimedia signal 302 is not encrypted by, for example, theBroadcaster 120 (FIG. 1), the Decryptor/Encryptor module 315 is notactivated and the multimedia signal 302 is forwarded directly to theDecoder/Encoder 320.

The Decoder/Encoder 320 detects the multimedia signal 302, for example,by decoding and decompressing the multimedia signal 302, so as toprovide a base composite signal, including an uncompressed video imagesignal and/or audio signal. The Decoder/Encoder 320 may include videoand audio codecs capable of decoding and/or encoding signals having thefollowing exemplary formats: ISO/IEC: MPEG-1, MPEG-1 Layer III (known asMP3), MPEG-1 Layer II, MPEG-2, MPEG-4, MPEG-4/AVC, MC; HE-AAC; ITU-T:H.261, H.262, H.263, H.264, G.711, G.722, G.722.1, G.722.2, G.723.1,G.726, G.728, G.729, G.729.1, G729a; SMPTE: VC-1; AVS; Dirac; Indeo;MJPEG; RealVideo; Theora; VP7; WMV Audio; AC3; ATRAC; FLAC; iLBC;Monkey's Audio; Musepack; RealAudio; SHN; Speex; Vorbis; WavPack; WMAImage compression formats: ISO/IEC/ITU-T: JPEG, JPEG 2000, JPEG-LS,JBIG, JBIG2; BMP; GIF; ILBM; PCX; PNG; TGA; TIFF; WMP Media containerformats 3GP; ASF; AVI; FLV; Matroska; MP4; MXF; NUT; Ogg; Ogg Media;QuickTime; RealMedia; AIFF; AU; and WAV. The base composite signal isforwarded to the Format Converter 325 and the Memory module 330.

The Format Converter 325 is configured to convert from the image/audioformat to a base composite signal, including a video image field signaland/or audio signal, and to convert from a base composite signal to theimage/audio formats, e.g., as identified above, as the skilled artisanwill readily recognize and comprehend. The resultant base compositesignal includes a video image field signal and/or an audio signal, wherethe audio signal may be the sound signal corresponding to the imagefield signal. As a result, information may be obtained from devices 205to 225 (shown in FIG. 2), which may provide for different formats (suchas, for example, those mentioned above), other than, for example, anMPEG video format used by display 110 (shown in FIG. 1). Moreover,information may be forwarded to devices 205 to 225 after conversion ofthe base composite signals by the Format Converter 325 to a formatuseable by devices 205 to 225, as is known in the art.

The Memory 330 stores decoded signals decoded by Decoder/Encoder module320, as well as base composite signals output by Format Converter module325. The Memory 330 may serve as a temporary storage for working dataused by the Processor 255.

The Image Extractor module 340 analyzes the composite signals stored inthe Memory 330. In accordance with instructions received from theController 335, the Image Extractor 340 determines which portions of animage field component, for example, of a base composite signal toextract. The extraction determination is made based on, for example,image recognition techniques and/or edge detection of the image fieldsignal, as is well known by those skilled in the image processingtechnologies.

In one non-limiting example according to an aspect of the invention, theImage Extractor 340 may extract the face of an Actor in a televisioncommercial for an automobile. The Image Extractor 340 transforms theextracted portion of the image field—in this example, the Actor'sface—from the time domain to the frequency domain using, for example, acosine transform, a Fourier transform, or any other well known imageprocessing transform, to provide a histogram of the extracted portion(e.g., the Actor's face).

The Feature Extractor 345 receives the extracted portion of the imagefield and the corresponding histogram for each image field signal. TheFeature Extractor 345 derives a plurality of motion vectors for theextracted portion of the image field through, for example, phasecorrelation between two or more image fields, or any other image motionvector generation scheme as is known in the art. The derived motionvectors provide, for example, movement information regarding thedifferent parts of the extracted portion of the image field. Forexample, in the above mentioned automobile advertisement, the FeatureExtractor 345 would identify the movement vectors pertaining to, forexample, the movement of the Actor's lips, eyes, facial muscles,eyebrows, etc. The Feature Extractor 345 outputs the generated motionvectors for each field.

Under control of the Controller 335, the output motion vectors from theFeature Extractor 345 are correlated to the corresponding extractedportions and histograms output from the Image Extractor 340 for eachimage field. The motion vectors and the corresponding extracted portionsand histograms are stored in the Memory 330. However, the skilledartisan will readily appreciate that the image portion extraction andhistogram generation of the Image Extractor 340 may be correlated to themotion vector generation of the Feature Extractor 345 withoutinstructions from the Controller 335.

The Image Processor 355 receives the original image field signal, thecorresponding extracted portions, the corresponding histogram, and thecorresponding motion vectors for the original image field signal fromthe Memory 330. However, the Image Processor 355 may receive thecorresponding extracted portions, the corresponding histogram, and themotion vectors directly from the Image Extractor 340 and the FeatureExtractor 345. The Image Processor 355 further retrieves athree-dimensional model image from the Memory 330 and/or the Database245 (shown in FIG. 2). The Image Processor 355 extracts a portion of thethree-dimensional model image to provide a model insertion image signal,which will be inserted into the composite signal in lieu of or as anoverlay of the extracted portion.

In alternative embodiments, the Image Processor 355 determines the modelinsertion image signal in at least one of two ways. However, the skilledartisan will readily recognize that other methodologies may be equallyemployed without departing from the scope and/or spirit of theinvention.

For example, an image azimuth angle signal Θ and an image altitude anglesignal φ may be included in each image field signal for each portion tobe extracted from the image field. The image azimuth and altitude anglesignals jointly referred to as the composite angle signal, which includecoordinates Θ and φ) would allow, for example, the Image Processor 355to determine that the extracted portion (such as, the Actor's face inthe automobile advertisement mentioned above) is a direct (φ=0 degrees),front angle (Θ=0 degrees) camera shot of the Actor's face. The compositeangle signal (Θ, φ) may be provided by the Broadcaster 120 (shown inFIG. 1), for example, in the blanking interval or vestigial sidebands inthe case of an analog signal, or in the extended data packets in thecase of a digital signal.

Alternatively, the Image Processor 355 may be configured to analyze anextracted portion for each image field and identify the azimuth andaltitude angles of the extracted portion. For example, the Processor 355may analyze the extracted portions and corresponding histograms outputfrom Image Extractor 340, and the extracted motion vectors output by theFeature Extractor 345 over a plurality of image fields to determine, forexample, a distance between an Actor's eyes relative to the Actor'snose, as well as a distance between the Actor's eyes and the Actor'shair-line and/or a plurality of points along a perimeter of the Actor'shead.

Once the model insertion image has been identified, the scale of theidentified model insertion image may be adjusted by the Image Processor355, so as to coincide with the scale of the extracted portion.Moreover, the model insertion image may be adjusted in terms of displaycharacteristics such as hue, saturation color space, contrast,brightness, color temperature, and the like, to match similar attributesof the extracted portion. Further, the surrounding portions of theoriginal image field, adjacent to the extracted portion, are alsoanalyzed in terms of the above display characteristics and the resultantdata may be used to further modify the identified model insertion imageso as to provide proper blending of the identified model insertion imageinto the original image field.

The Image Processor 355 further determines feature adjustment in theextracted portion based on, for example, the motion vectors provided byFeature Extractor 345. Particularly, Image Processor 355 adjusts partsof the identified model image to match similar parts in the extractedportion. For example, in the above automobile advertisement example,where the face of an Actor has been extracted by Image Extractor 340,the Image Processor will determine, for example, movement of the Actor'seyes and lips based on the motion vectors provided by Feature Extractor345 and apply similar motion vectors to the identified model image(i.e., the image to be inserted). Thus, in the automobile advertisementexample, an image of the user's face, for example, will be inserted intothe original image fields such that the user's face will replace theActor's face, including the specific gestures, body language, and otherattributes.

Referring to FIG. 4, an exemplary embodiment is provided for capturingone or more images for insertion by the CCAM 200, according to an aspectof the invention. For simplicity of explanation, the embodiment is shownto include planar-movement of an image capture device 420 in thetwo-dimensional X-Z plane, where the azimuth angle (Θ) varies, but thealtitude angle (φ) is fixed at 0 degrees. However, the skilled artisanwill readily recognize and appreciate that the image capture device 420may also be moved in the third-dimension, where the altitude angle φvaries in a range between minus 90 degrees and plus 90 degrees so as togenerate a complete three-dimensional image model of the subject 405.

According to the depicted embodiment, the subject 405 is positioned atthe center of a circular orbit 410 (or spherical orbit 410 in the caseof three-dimensional image capture). The three-dimensional coordinatesof the subject 405 relative to the position of image capture device 420are determined and recorded through known coordinate-registrationsystems such as, for example, multi-point-touch registration, or by anyother method capable of accurately registering a three dimensionalposition of one or more points on the surface of subject 405 relative tothe image capture device 420. For example, as shown in FIG. 6, the imagecapture device 420 may be registered relative to the subject 405 bytouching the subject 405 with an image registration device (not shown),included in the image capture device 420, or provided separately, at oneor more of points 600 and recording the relative coordinates (x, y, z)of the one or more touched points 600.

The image capture device 420 is initialized at an arbitrary positionsuch as, for example, 420A in FIG. 4. The three-dimensional coordinatescorresponding to point 420A of the image capture device 420 are recordedfor the initial position 420A, as well as the relationship between theimage capture device 420 position and the position of subject 405.Moreover, the perspective angle of image capture is recorded for point420A—in the non-limiting example shown in FIG. 4, the angle is 270degrees, representing the right profile perspective image of the subject405. The image capture device 420 may be moved through n discretepositions, where n is a whole number integer greater than or equal toone so as to capture various perspectives of the subject 405. Theplurality of captured images, including positional information andrespective perspective angles of image capture relative to the subject405 are recorded such that a three-dimensional image may be reproduced.

For example, referring to FIGS. 5A through 5F, six exemplary,non-limiting images (n=6) are shown that may be captured for the sixrespective positional points 420A to 420F (in FIG. 4) of the imagecapture device 420. For example, the image of the subject 405 capturedfrom an initial position 420A of the image capture device 420 isdepicted in FIG. 5A, which corresponds to a 270 degree offset from thereference axis (0 degrees) of a direct view of the subject's face image(shown, for example, in FIG. 5C). A second image of the subject 405captured from the position 420B of the image capture device 420 isdepicted in FIG. 5B, which corresponds to a 315 degree offset from thereference axis. A third image of the subject 405 captured from theposition 420C of the image capture device 420 is depicted in FIG. 5C,which corresponds to the reference axis (0 degrees). A fourth image ofthe subject 405 captured from the position 420D of the image capturedevice 420 is depicted in FIG. 5D, which corresponds to a 45 degreeoffset from the reference axis. A fifth image of the subject image 405captured from the position 420E of the image capture device 420 isdepicted in FIG. 5E, which corresponds to a 90 degree offset from thereference axis. And, a sixth image of the subject 405 captured from theposition 420F of the image capture device 420 is depicted in FIG. 5F,which corresponds to a 180 degrees offset from the reference axis.

The multiple captured images (e.g., FIGS. 5A to 5F) of the subject 405,including at least one image of the face of subject 405 (FIG. 5C), areprocessed and combined to create a three-dimensional image of thesubject 405, as the skilled artisans will readily appreciate. Thethree-dimensional image of the subject 405, including each of thediscrete images captured from the positions 420A to 420F in FIG. 4, aretemporarily stored, referring to FIG. 2, in the CCAM 200 working memory,RAM 235, and subsequently recorded in Database 245 for subsequentretrieval.

A process that may be executed by CCAM 200, according to an aspect ofthe invention, will be described below, with reference to FIG. 7.Moreover, a computer readable medium is provided having recorded thereoncode segments for each of the below described processes.

Referring to FIG. 7, the exemplary process begins (step S705) atinitialization of the CCAM 200 and determines that image insertion hasbeen selected by the user. The process determines whether one or morespecific categories of source signals have been selected by the user(step S715). The categories may include, for example, various categoriesof television commercials. Hence, the user may select specificcategories, including, but not limited to, advertisements forautomobiles, hair products, men's clothing, men's accessories, women'sclothing, women's accessories, baby products, pet products, etc. Thevarious categories may be determined in a variety of ways, withoutdeparting from the spirit and/or scope of the invention. For example,the Broadcaster 120 may determine the categories of source signals,grouping advertisements according to the advertisers' instructions orits own cataloging systems. Alternatively, the users may create theirown categories based on individual preferences.

If the process determines that a specific category of source signals hasnot been selected by the user (“NO” at step S715), the process queriesthe user for one or more specific categories (step S720). The query maybe, for example, an interactive session provided via the display 110(shown in FIG. 1) over a communication link such as, for example, theNetwork 115 (shown in FIG. 1) where the user is provided with a list ofcategories of source signals the user may select for image insertion.

The Settop box 105 may include an input device such as, for example, akeyboard (not shown), an interactive voice control module, or the like,by which the user may input information, including user selections.Further, the user may be provided with a telephone number or URL addresswhich the user may contact and input selections via existing telephonehardware or computer hardware as is known in the related arts.

Alternatively, rather than an interactive session as described above, orin addition to the interactive session, the process may cause a surveyor list to be mailed via a physical medium (such as, for example, UnitedStates Postal Service mail) or electronic medium (such as, for example,email, text messaging, instant messaging, etc.) to the user. The surveyor list may include various types or categories of televisionadvertisements that are commonly broadcast by, for example, Broadcaster120 (shown in FIG. 1). The user would be invited to select thosecategories of advertisements within which the user desires to insert auser selected/defined image.

In addition to the survey or list, a questionnaire may be providedrequesting user feedback and/or comments. The questionnaire may includequestions directed to determining what type of further advertising theuser would like to receive.

In an alternative embodiment, according to an aspect of the invention,the CCAM 200 may be integrated in the Broadcaster 120 rather than theSettop Box 105 (shown in FIG. 1). In this instance, a questionnaire maybe provided to a user offering customized commercial advertisement. Inaddition to the offer, the questionnaire may request the user to selectone or more categories of advertisements in which the user would like toinsert a user selected image. The user would further be requested toprovide a series of images (for example, shown in FIGS. 5A to 5F),discussed below with respect to step S785.

Returning to FIG. 7, upon completion of the interactive session with theuser (step S720), for example, the process receives the user'sselections of specific categories (step S725). The received user'sselections (and images if applicable) are then stored in a user profilefile (step S730), e.g., in the Database 245 (FIG. 2) and/or the Database125 (FIG. 1), as discussed above. In addition to specific categories ofadvertisements desired to be modified to insert user defined images, theuser profile file may include further information such as, user billinginformation; services provided to user; specific images that may beinserted; one or more look up tables listing stored images and types ofadvertisements in which the stored images may be inserted; user address,including mail and email addresses; user telephone number; user nameand/or identification information; and/or one or more user definedavatars that may be inserted into the selected categories ofadvertisements.

The process then proceeds to determine whether one or more images forinsertion have been stored (step S735). Likewise, If the processdetermines that a specific category of source image signal has beenidentified by the user (“YES” at step S715), the process determineswhether one or more images have been stored (step S735).

If it is determined that one or more images are not stored (“NO” at stepS735), the process queries the user through a user interface, such as,for example, the display 110 in FIG. 1, and obtains one or more imagesfrom a source device (step S785). The Settop box 105 may store thereceived one or more images in a portion of the Database 245, forexample. The Settop box 105, via CCAM 200, inserts the user's selectedimages into the user defined categories of advertisements at the usersite. The selected images may be inserted on a real-time basis asbroadcast signals are received, or on an a priori basis where theselected images are inserted into advertisements and stored in Database245, for example.

Alternatively, as discussed above, the query may occur through aninteractive session with the user via the Network 115, on the user'sdisplay 110 (FIG. 1), or the query may be sent (via physical orelectronic mail) to the user in the form of a survey, list and/orquestionnaire, as discussed above with respect to step S725. TheBroadcaster 120 may store the received series of images in a portion ofthe Database 125, for example, dedicated to the user. The Broadcaster120 may insert the user's selected images into the user definedcategories of advertisements at the transmitter end so that the userwould receive, at the Settop Box 105, a multimedia signal that includesa modified image signal, including an inserted image. The selectedimages may be inserted on a real-time basis as broadcast signals areprepared for broadcast, or on an a priori basis where the selectedimages are inserted into advertisements, for example, and stored inDatabase 125.

The obtained one or more images are then processed to generate athree-dimensional model image (step S788), as discussed, for example,with respect to FIGS. 4-6. The three-dimensional model image, includingthe one or more images, is stored (step S790). Once the model image,including the one or more images, is stored (step S790), e.g., in theDatabase 245 (FIG. 2) and/or the Database 125 (FIG. 1), the processreturns to the preceding inquiry, where a determination is made whetheran image model has been stored (step S735).

If it is determined that one or more images are stored (“YES” at stepS735), for example, at the Settop Box 105 or the Broadcaster 120, thenthe process receives a multimedia signal that includes a source imagesignal of the category identified by the user, decoded and convertedinto a base composite signal that includes an image field signal (stepS740). In an embodiment, the multimedia signal may also need to bedecrypted to obtain the base component signal. The image field signal isthen analyzed, and one or more portions of the image field signal areextracted based on the analysis (step S745). The extracted one or moreportions are transformed from the time domain to the frequency domain,and a histogram is generated for each of the extracted portions (stepS748). The extracted one or more portions are further analyzed andfeatures are extracted from the one or more extracted portions,generating motion vectors for the one or more extracted portions (stepS750).

The process retrieves a user defined and selected model image (stepS755), for example, stored at step S790. One or more portions of theretrieved model image are identified (step S760). The identified one ormore portions of the model image are processed to adjust the attributesof the identified one or more portions of the model image, theattributes including, for example, edge-blending effect, scale, hue,brightness, color saturation, etc. (step S765). The processed one ormore model image portions are inserted into the received image fieldsignal in place of the one or more extracted portions, or as overlayonto the received image field signal (step S770). The process may insertor overlay the processed one or more model image portions into thereceived image field signal based on a received mode signal, whichindicates selection of an insert mode, overlay mode, or a composite ofinsert and overlay modes.

The process determines if image insertion is completed (step S775). Ifimage insertion is complete (“YES” at step S775), the modified image isdisplayed (step S780), for example, on the display 110, and the processends (step S795). However, if the process determines that imageinsertion is not complete (“NO” at step S775), the process returns toreceive further image field signals (step S740).

In this manner, a user is able to select a category of advertisements orother source media, which the user would like to modify withpersonalized images, sound, or other type of indicia. The selectedcategory of advertisements may be modified at the user site or at thebroadcaster site, thereby providing reproduction of personalizedmultimedia signals.

Although the above disclosure has been provided with examples ofmodifying multimedia signals to insert and/or overlay user-definedimages, the skilled artisan will readily appreciate that multimediasignals may also be modified to insert and/or overlay sound signals,control signals, and like, without departing from the scope and/orspirit of the invention. For example, in addition or alternatively toprocessing video signals as discussed above, vowels, words, phrases,and/or other sounds may be implemented in modifying the multimediasignals to provide personalized audio portions of multimedia signals. Inthis regard, for example, a user may store vowels, words, phrases,and/or other sounds and the Processor 255 (in FIG. 2) may generate oneor more signature files based on the stored sounds to generate soundsignals to be inserted or overlayed in the received multimedia signal,which may be in the form of a radio broadcast signal.

Further, the apparatus, method and/or computer readable medium discussedabove may include modifying received radio programs and/oradvertisements by inserting and/or overlaying user-defined soundsignals. In this regard, the received radio programs may be broadcast bysatellite radio broadcast entities.

Further, the apparatus, method and/or computer readable medium discussedabove may include modifying received video and/or sound signals receivedfrom a computer game device, which may be a standalone computer or aremote computer connected via a network (such as, for example, a localarea network, a wide area network, or the like), to insert and/oroverlay user-defined image and/or sound signals. In this regard, userdefined objects, which may include video and/or audio, may be insertedand/or overlayed into a received multimedia signal from the computergame device such that a displayed video game will include user definedobjects. For example, a computer game user may insert or overlay animage of his own face and/or the user's own voice instead of the faceand/or voice of a character in a computer game.

Further, the term “insert,” or any derivative thereof, as used inherein, is meant to also include “overlay,” or any derivative thereof,and/or “blend,” or any derivative thereof.

Further, the apparatus, method and/or computer readable medium discussedabove may include user-specific and/or account-specific customizedmodification information. For example, a plurality of profiles may beprovided that include parameters specific to one or more users and/oruser accounts. Each profile may contain one or more of the followingparameters for personalized modification of advertisements, or othersource media. The parameters may include, for example, a password toaccess the user profile and/or activate personalized image insertion; atleast one selected category of advertisement, or other source media;billing information specific to the user and/or user account; servicesprovided to the specific user and/or user account; specific images thatmay be inserted for the specific user and/or user account; one or morelook up tables listing stored images and types of advertisements inwhich the stored images may be inserted for the specific user and/oruser account; user address, including mail and email addresses for thespecific user and/or user account; at least one telephone number for thespecific user and/or user account; a user name and/or identificationinformation, including biometric information such as, for example,fingerprint, retinal print, and the like; and/or one or more userdefined avatars that may be inserted into the selected categories ofadvertisements for the specific user and/or user account.

Further, the invention may be used in systems including, for example, aserver-client relationship and/or peer-to-peer relationship. Theinvention may be used by service providers such as, for example,Internet service providers so as to enable users to personalizeadvertisements that may be reproduced at the terminals of the users,including such advertisements as, for example, pop-advertisements, andthe like.

Further, the invention may be used by program broadcasters, includingtelevision and radio broadcasters and/or other service providers, tomodify transmitted programs such as, for example, television shows,including game shows, “sit-coms,” “made-for-television movies,” withpersonalized images and/or sounds. For example, user-defined multimediasignals, including images and/or sounds, may be sent to a television orradio broadcaster, which then uses the user-defined multimedia signalsto modify a transmitted game show so as to insert or overlay theuser-defined multimedia signal, including images, sounds and/or otherindicia, to alter or customize the appearance or voice of thecontestants and/or host.

The selected category of advertisements may be modified at a user site.Personalized information provided by a user may be stored, retrieved,processed and inserted or overlaid at the user site into receivedmultimedia signals to provide reproduction of personalized multimediasignals.

Further, the selected category of advertisements may be modified at abroadcaster site. In that case, personalized information may be providedfrom a user to a broadcaster via a communication link, includingphysical or electronic communication media. The personalized informationmay be stored, retrieved, processed and inserted or overlaid at thebroadcaster site into multimedia signals that are transmitted, or may betransmitted at some later time to one or more user sites.

Although the invention has been described with reference to severalexemplary embodiments, it is understood that the words that have beenused are words of description and illustration, rather than words oflimitation. Changes may be made within the purview of the appendedclaims, as presently stated and as amended, without departing from thescope and spirit of the invention in its aspects. Although the inventionhas been described with reference to particular means, materials andembodiments, the invention is not intended to be limited to theparticulars disclosed; rather, the invention extends to all functionallyequivalent structures, methods, and uses such as are within the scope ofthe appended claims.

In accordance with various embodiments of the present invention, themethods described herein are intended for operation as software programsrunning on a computer processor. Dedicated hardware implementationsincluding, but not limited to, application specific integrated circuits,programmable logic arrays and other hardware devices can likewise beconstructed to implement the methods described herein. Furthermore,alternative software implementations including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods described herein.

It should also be noted that the software implementations of the presentinvention as described herein are optionally stored on a tangiblestorage medium, such as: a magnetic medium such as a disk or tape; amagneto-optical or optical medium such as a disk; or a solid statemedium such as a memory card or other package that houses one or moreread-only (non-volatile) memories, random access memories, or otherre-writable (volatile) memories. A digital file attachment to e-mail orother self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. Accordingly, the invention is considered to include a tangiblestorage medium or distribution medium, as listed herein and includingart-recognized equivalents and successor media, in which the softwareimplementations herein are stored.

Although the present specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the invention is not limited to such standards andprotocols. Accordingly, replacement standards and protocols having thesame functions are considered equivalents.

1. An object insertion apparatus for inserting a model object into atleast one program, the apparatus comprising: a memory for storing themodel object; a selector for receiving selection of the model object anda category of program, the category of program comprising the at leastone program; and an inserter for inserting the model object into the atleast one program based on the selection of the category of program. 2.The apparatus according to claim 1, wherein the apparatus is located ata broadcaster site that transmits the at least one program to a usersite.
 3. The apparatus according to claim 2, further comprising: acommunicator for communicating with the user site, the communicatorreceiving the selected model object and the category of program from theuser site.
 4. The apparatus according to claim 3, the insertercomprising: an object extractor for extracting a portion of an objectfield signal from the at least one program; a feature extractor forextracting a feature from the extracted portion of the object fieldsignal; and an object processor that modifies the model object based onthe extracted portion of the object field signal and the extractedfeature from the extracted portion of the object field signal, whereinthe object processor is adapted to insert the modified model object inthe object field signal.
 5. The apparatus according to claim 4, whereinthe feature extractor generates at least one motion vector for theextracted portion of the object field signal.
 6. The apparatus accordingto claim 4, further comprising: a decryptor for decrypting a multimediasignal representing the program; a decoder for decoding the decryptedmultimedia signal; and a format converter for converting the decodedmultimedia signal to a base composite signal, the base composite signalcomprising the object field signal.
 7. The apparatus according to claim1, wherein the model object comprises an image, and the at least oneprogram comprises an advertisement.
 8. The apparatus according to claim1, wherein the model object comprises a sound, and the at least oneprogram comprises an advertisement.
 9. The apparatus according to claim4, further comprising: an encryptor for encrypting a base compositesignal, comprising the object field signal; an encoder for encoding thebase composite signal; and a format converter for converting the basecomposite signal to a multimedia signal.
 10. An object insertion methodfor inserting a selected object into at least one program, the at leastone program being associated with one of a plurality of categories ofprograms, the method comprising: receiving a selection of a category ofprogram from the plurality of categories of programs, the selectedcategory of program comprising the at least one program; receiving aselection of a model object from one or more stored model objects; andinserting the selected model object in the at least one program based onthe selected category of program.
 11. The method according to claim 10,wherein the model object comprises an image, and the at least oneprogram comprises an advertisement.
 12. The method according to claim10, wherein the inserting is performed at a broadcaster site thattransmits the at least one program to a user site.
 13. The methodaccording to claim 12, further comprising: receiving a selection of atleast one of an insert mode or an overlay mode.
 14. The method accordingto claim 12, the receiving of at least one of the category of program orthe model image comprising receiving a survey or questionnaire from asubscriber via physical mail or electronic mail.
 15. The methodaccording to claim 14, wherein the inserting comprises: extracting aportion of an object field signal of the at least one program;extracting one or more features from the extracted portion of the objectfield signal; modifying the selected model object based on the extractedportion of the object field signal and the extracted one or morefeatures from the extracted portion of the object field signal; andinserting the modified model object in the object field signal.
 16. Themethod according to claim 10, wherein the model object comprises asound, and the at least one program comprises an advertisement.
 17. Themethod according to claim 15, further comprising: generating one or moremotion vectors for the extracted portion of the object field signal. 18.A computer readable medium for storing a program that inserts a selectedobject into at least one program, the at least one program beingassociated with one of a plurality of categories of programs, the mediumcomprising: a category selection receiving code segment for receivingselection of a category of program from the plurality of categories ofprograms, the selected category of program comprising the at least oneprogram; a model object selection receiving code segment for receivingselection of a model object from one or more stored model objects basedon the selected category of program; an inserting code segment forinserting the selected model object in the at least one program
 19. Themedium according to claim 18, further comprising: a mode receiving codesegment for receiving selection of at least one of an insert mode or anoverlay mode.
 20. The medium according to claim 18, further comprising:a portion extracting code segment for extracting a portion of an objectfield signal of the at least one program; a feature extracting codesegment for extracting a feature from the extracted portion of theobject field signal; and a modifying code segment for modifying theselected model object based on the extracted portion of the object fieldsignal and the extracted feature from the extracted portion of theobject field signal.